{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qFdPvlXBOdUN"
      },
      "source": [
        "# Estimators"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/guide/estimator\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/guide/estimator.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/guide/estimator.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/guide/estimator.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oEinLJt2Uowq"
      },
      "source": [
        "This document introduces `tf.estimator`—a high-level TensorFlow\n",
        "API. Estimators encapsulate the following actions:\n",
        "\n",
        "* training\n",
        "* evaluation\n",
        "* prediction\n",
        "* export for serving\n",
        "\n",
        "You may either use the pre-made Estimators we provide or write your\n",
        "own custom Estimators.  All Estimators—whether pre-made or custom—are\n",
        "classes based on the `tf.estimator.Estimator` class.\n",
        "\n",
        "For a quick example try [Estimator tutorials](../../tutorials/estimator/linear.ipynb). For an overview of the API design, see the [white paper](https://arxiv.org/abs/1708.02637)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Wg5zbBliQvNL"
      },
      "source": [
        "## Advantages\n",
        "\n",
        "Similar to a `tf.keras.Model`, an `estimator` is a model-level abstraction. The `tf.estimator` provides some capabilities currently still under development for `tf.keras`. These are:\n",
        "\n",
        "  * Parameter server based training\n",
        "  * Full [TFX](http://tensorflow.org/tfx) integration."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yQ8fQYt_VD5E"
      },
      "source": [
        "## Estimators Capabilities\n",
        "Estimators provide the following benefits:\n",
        "\n",
        "* You can run Estimator-based models on a local host or on a distributed multi-server environment without changing your model. Furthermore, you can run Estimator-based models on CPUs, GPUs, or TPUs without recoding your model.\n",
        "* Estimators provide a safe distributed training loop that controls how and when to:    \n",
        "    * load data\n",
        "    * handle exceptions\n",
        "    * create checkpoint files and recover from failures\n",
        "    * save summaries for TensorBoard\n",
        "\n",
        "When writing an application with Estimators, you must separate the data input\n",
        "pipeline from the model.  This separation simplifies experiments with\n",
        "different data sets."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sXNBeY-oVxGQ"
      },
      "source": [
        "## Pre-made Estimators\n",
        "\n",
        "Pre-made Estimators enable you to work at a much higher conceptual level than the base TensorFlow APIs. You no longer have to worry about creating the computational graph or sessions since Estimators handle all the \"plumbing\" for you. Furthermore, pre-made Estimators let you experiment with different model architectures by making only minimal code changes.  `tf.estimator.DNNClassifier`, for example, is a pre-made Estimator class that trains classification models based on dense, feed-forward neural networks.\n",
        "\n",
        "### Structure of a pre-made Estimators program\n",
        "\n",
        "A TensorFlow program relying on a pre-made Estimator typically consists of the following four steps:\n",
        "\n",
        "#### 1. Write one or more dataset importing functions.\n",
        "\n",
        "For example, you might create one function to import the training set and another function to import the test set. Each dataset importing function must return two objects:\n",
        "\n",
        "* a dictionary in which the keys are feature names and the values are Tensors (or SparseTensors) containing the corresponding feature data\n",
        "* a Tensor containing one or more labels\n",
        "\n",
        "For example, the following code illustrates the basic skeleton for an input function:\n",
        "\n",
        "```\n",
        "def input_fn(dataset):\n",
        "    ...  # manipulate dataset, extracting the feature dict and the label\n",
        "    return feature_dict, label\n",
        "```\n",
        "\n",
        "See [data guide](../../guide/data.md) for details.\n",
        "\n",
        "#### 2. Define the feature columns.\n",
        "\n",
        "Each `tf.feature_column` identifies a feature name, its type, and any input pre-processing. For example, the following snippet creates three feature columns that hold integer or floating-point data. The first two feature columns simply identify the feature's name and type. The third feature column also specifies a lambda the program will invoke to scale the raw data:\n",
        "\n",
        "```\n",
        "# Define three numeric feature columns.\n",
        "population = tf.feature_column.numeric_column('population')\n",
        "crime_rate = tf.feature_column.numeric_column('crime_rate')\n",
        "median_education = tf.feature_column.numeric_column(\n",
        "  'median_education',\n",
        "  normalizer_fn=lambda x: x - global_education_mean)\n",
        "```\n",
        "For further information, see the [feature columns tutorial](https://www.tensorflow.org/tutorials/keras/feature_columns).\n",
        "\n",
        "#### 3. Instantiate the relevant pre-made Estimator.\n",
        "\n",
        "For example, here's a sample instantiation of a pre-made Estimator named `LinearClassifier`:\n",
        "\n",
        "```\n",
        "# Instantiate an estimator, passing the feature columns.\n",
        "estimator = tf.estimator.LinearClassifier(\n",
        "  feature_columns=[population, crime_rate, median_education])\n",
        "```\n",
        "For further information, see the [linear classifier tutorial](https://www.tensorflow.org/tutorials/estimator/linear).\n",
        "\n",
        "#### 4. Call a training, evaluation, or inference method.\n",
        "\n",
        "For example, all Estimators provide a `train` method, which trains a model.\n",
        "\n",
        "```\n",
        "# `input_fn` is the function created in Step 1\n",
        "estimator.train(input_fn=my_training_set, steps=2000)\n",
        "```\n",
        "You can see an example of this below.\n",
        "\n",
        "### Benefits of pre-made Estimators\n",
        "\n",
        "Pre-made Estimators encode best practices, providing the following benefits:\n",
        "\n",
        "* Best practices for determining where different parts of the computational graph should run, implementing strategies on a single machine or on a\n",
        "    cluster.\n",
        "*   Best practices for event (summary) writing and universally useful\n",
        "    summaries.\n",
        "\n",
        "If you don't use pre-made Estimators, you must implement the preceding features yourself."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oIaPjYgnZdn6"
      },
      "source": [
        "## Custom Estimators\n",
        "\n",
        "The heart of every Estimator—whether pre-made or custom—is its *model function*, which is a method that builds graphs for training, evaluation, and prediction. When you are using a pre-made Estimator, someone else has already implemented the model function. When relying on a custom Estimator, you must write the model function yourself.\n",
        "\n",
        "## Recommended workflow\n",
        "\n",
        "1. Assuming a suitable pre-made Estimator exists, use it to build your first model and use its results to establish a baseline.\n",
        "2. Build and test your overall pipeline, including the integrity and reliability of your data with this pre-made Estimator.\n",
        "3. If suitable alternative pre-made Estimators are available, run experiments to determine which pre-made Estimator produces the best results.\n",
        "4. Possibly, further improve your model by building your own custom Estimator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "l0QKHuEJ4Kc_"
      },
      "outputs": [],
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "kRr7DGZxFApM"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "import tensorflow as tf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "IqR2PQG4ZaZ0"
      },
      "outputs": [],
      "source": [
        "import tensorflow_datasets as tfds\n",
        "tfds.disable_progress_bar()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "P7aPNnXUbN4j"
      },
      "source": [
        "## Create an Estimator from a Keras model\n",
        "\n",
        "You can convert existing Keras models to Estimators with `tf.keras.estimator.model_to_estimator`. Doing so enables your Keras\n",
        "model to access Estimator's strengths, such as distributed training.\n",
        "\n",
        "Instantiate a Keras MobileNet V2 model and compile the model with the optimizer, loss, and metrics to train with:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "XE6NMcuGeDOP"
      },
      "outputs": [],
      "source": [
        "keras_mobilenet_v2 = tf.keras.applications.MobileNetV2(\n",
        "    input_shape=(160, 160, 3), include_top=False)\n",
        "\n",
        "estimator_model = tf.keras.Sequential([\n",
        "    keras_mobilenet_v2,\n",
        "    tf.keras.layers.Flatten(),\n",
        "    tf.keras.layers.Dense(1, activation='softmax')\n",
        "])\n",
        "\n",
        "# Compile the model\n",
        "estimator_model.compile(\n",
        "    optimizer='adam',\n",
        "    loss='binary_crossentropy',\n",
        "    metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "A3hcxzcEfYfX"
      },
      "source": [
        "Create an `Estimator` from the compiled Keras model. The initial model state of the Keras model is preserved in the created `Estimator`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "UCSSifirfyHk"
      },
      "outputs": [],
      "source": [
        "est_mobilenet_v2 = tf.keras.estimator.model_to_estimator(keras_model=estimator_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8jRNRVb_fzGT"
      },
      "source": [
        "Treat the derived `Estimator` as you would with any other `Estimator`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Rv9xJk51e1fB"
      },
      "outputs": [],
      "source": [
        "IMG_SIZE = 160  # All images will be resized to 160x160\n",
        "\n",
        "def preprocess(image, label):\n",
        "  image = tf.cast(image, tf.float32)\n",
        "  image = (image/127.5) - 1\n",
        "  image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))\n",
        "  return image, label"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Fw8OjwujVBkc"
      },
      "outputs": [],
      "source": [
        "def train_input_fn(batch_size):\n",
        "  data = tfds.load('cats_vs_dogs', as_supervised=True)\n",
        "  train_data = data['train']\n",
        "  train_data = train_data.map(preprocess).shuffle(500).batch(batch_size)\n",
        "  return train_data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JMb0cuy0gbTi"
      },
      "source": [
        "To train, call Estimator's train function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "4JsvMp8Jge80"
      },
      "outputs": [],
      "source": [
        "est_mobilenet_v2.train(input_fn=lambda: train_input_fn(32), steps=500)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jvr_rAzngY9v"
      },
      "source": [
        "Similarly, to evaluate, call the Estimator's evaluate function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "kVNPqysQgYR2"
      },
      "outputs": [],
      "source": [
        "est_mobilenet_v2.evaluate(input_fn=lambda: train_input_fn(32), steps=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5HeTOvCYbjZb"
      },
      "source": [
        "For more details, please refer to the documentation for `tf.keras.estimator.model_to_estimator`."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "Tce3stUlHN0L"
      ],
      "name": "estimator.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true,
      "version": "0.3.2"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
